{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Basics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7f35ea8a2630>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "torch.manual_seed(1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Scalar is a single number.\n",
    "* Vector is an array of numbers.\n",
    "* Matrix is a 2-D array of numbers.\n",
    "* Tensors are N-D arrays of numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create tensors by specifying the shape as arguments.  Here is a tensor with 5 rows and 3 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def describe(x):\n",
    "    print(\"Type: {}\".format(x.type()))\n",
    "    print(\"Shape/size: {}\".format(x.shape))\n",
    "    print(\"Values: \\n{}\".format(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[ 3.1654e+09,  4.5635e-41, -5.4825e-21],\n",
      "        [ 3.0718e-41,  4.4842e-44,  0.0000e+00]])\n"
     ]
    }
   ],
   "source": [
    "describe(torch.Tensor(2, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[ 0.0461,  0.4024, -1.0115],\n",
      "        [ 0.2167, -0.6123,  0.5036]])\n"
     ]
    }
   ],
   "source": [
    "describe(torch.randn(2, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's common in prototyping to create a tensor with random numbers of a specific shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0.7749, 0.8208, 0.2793],\n",
      "        [0.6817, 0.2837, 0.6567]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(2, 3)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also initialize tensors of ones or zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.]])\n"
     ]
    }
   ],
   "source": [
    "describe(torch.zeros(2, 3))\n",
    "x = torch.ones(2, 3)\n",
    "describe(x)\n",
    "x.fill_(5)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors can be initialized and then filled in place. \n",
    "\n",
    "Note: operations that end in an underscore (`_`) are in place operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.FloatTensor\n",
      "torch.Size([3, 4])\n",
      "tensor([[5., 5., 5., 5.],\n",
      "        [5., 5., 5., 5.],\n",
      "        [5., 5., 5., 5.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.Tensor(3,4).fill_(5)\n",
    "print(x.type())\n",
    "print(x.shape)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors can be initialized from a list of lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[1., 2.],\n",
      "        [2., 4.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.Tensor([[1, 2,],  \n",
    "                  [2, 4,]])\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors can be initialized from numpy matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.DoubleTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0.6938, 0.0125, 0.7894],\n",
      "        [0.4493, 0.1734, 0.4403]], dtype=torch.float64)\n",
      "float64\n"
     ]
    }
   ],
   "source": [
    "npy = np.random.rand(2, 3)\n",
    "describe(torch.from_numpy(npy))\n",
    "print(npy.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tensor Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The FloatTensor has been the default tensor that we have been creating all along"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "x = torch.arange(6).view(2, 3)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1., 2., 3.],\n",
      "        [4., 5., 6.]])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6]])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[1., 2., 3.],\n",
      "        [4., 5., 6.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.FloatTensor([[1, 2, 3],  \n",
    "                       [4, 5, 6]])\n",
    "describe(x)\n",
    "\n",
    "x = x.long()\n",
    "describe(x)\n",
    "\n",
    "x = torch.tensor([[1, 2, 3], \n",
    "                  [4, 5, 6]], dtype=torch.int64)\n",
    "describe(x)\n",
    "\n",
    "x = x.float() \n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[ 1.5385, -0.9757,  1.5769],\n",
      "        [ 0.3840, -0.6039, -0.5240]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(2, 3)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[ 3.0771, -1.9515,  3.1539],\n",
      "        [ 0.7680, -1.2077, -1.0479]])\n"
     ]
    }
   ],
   "source": [
    "describe(torch.add(x, x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[ 3.0771, -1.9515,  3.1539],\n",
      "        [ 0.7680, -1.2077, -1.0479]])\n"
     ]
    }
   ],
   "source": [
    "describe(x + x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([6])\n",
      "Values: \n",
      "tensor([0, 1, 2, 3, 4, 5])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(6)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n"
     ]
    }
   ],
   "source": [
    "x = x.view(2, 3)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([3])\n",
      "Values: \n",
      "tensor([3, 5, 7])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2])\n",
      "Values: \n",
      "tensor([ 3, 12])\n"
     ]
    }
   ],
   "source": [
    "describe(torch.sum(x, dim=0))\n",
    "describe(torch.sum(x, dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([3, 2])\n",
      "Values: \n",
      "tensor([[0, 3],\n",
      "        [1, 4],\n",
      "        [2, 5]])\n"
     ]
    }
   ],
   "source": [
    "describe(torch.transpose(x, 0, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([1, 2])\n",
      "Values: \n",
      "tensor([[0, 1]])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([])\n",
      "Values: \n",
      "1\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "x = torch.arange(6).view(2, 3)\n",
    "describe(x)\n",
    "describe(x[:1, :2])\n",
    "describe(x[0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[0, 2],\n",
      "        [3, 5]])\n"
     ]
    }
   ],
   "source": [
    "indices = torch.LongTensor([0, 2])\n",
    "describe(torch.index_select(x, dim=1, index=indices))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [0, 1, 2]])\n"
     ]
    }
   ],
   "source": [
    "indices = torch.LongTensor([0, 0])\n",
    "describe(torch.index_select(x, dim=0, index=indices))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2])\n",
      "Values: \n",
      "tensor([0, 4])\n"
     ]
    }
   ],
   "source": [
    "row_indices = torch.arange(2).long()\n",
    "col_indices = torch.LongTensor([0, 1])\n",
    "describe(x[row_indices, col_indices])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Long Tensors are used for indexing operations and mirror the `int64` numpy type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6],\n",
      "        [7, 8, 9]])\n",
      "torch.int64\n",
      "int64\n"
     ]
    }
   ],
   "source": [
    "x = torch.LongTensor([[1, 2, 3],  \n",
    "                      [4, 5, 6],\n",
    "                      [7, 8, 9]])\n",
    "describe(x)\n",
    "print(x.dtype)\n",
    "print(x.numpy().dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can convert a FloatTensor to a LongTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[1, 2, 3],\n",
      "        [4, 5, 6],\n",
      "        [7, 8, 9]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.FloatTensor([[1, 2, 3],  \n",
    "                       [4, 5, 6],\n",
    "                       [7, 8, 9]])\n",
    "x = x.long()\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Special Tensor initializations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a vector of incremental numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(0, 10)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it's useful to have an integer-based arange for indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(0, 10).long()\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations\n",
    "\n",
    "Using the tensors to do linear algebra is a foundation of modern Deep Learning practices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reshaping allows you to move the numbers in a tensor around.  One can be sure that the order is preserved.  In PyTorch, reshaping is called `view`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n",
      "         18, 19]])\n",
      "tensor([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],\n",
      "        [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]])\n",
      "tensor([[ 0,  1,  2,  3,  4],\n",
      "        [ 5,  6,  7,  8,  9],\n",
      "        [10, 11, 12, 13, 14],\n",
      "        [15, 16, 17, 18, 19]])\n",
      "tensor([[ 0,  1,  2,  3],\n",
      "        [ 4,  5,  6,  7],\n",
      "        [ 8,  9, 10, 11],\n",
      "        [12, 13, 14, 15],\n",
      "        [16, 17, 18, 19]])\n",
      "tensor([[ 0,  1],\n",
      "        [ 2,  3],\n",
      "        [ 4,  5],\n",
      "        [ 6,  7],\n",
      "        [ 8,  9],\n",
      "        [10, 11],\n",
      "        [12, 13],\n",
      "        [14, 15],\n",
      "        [16, 17],\n",
      "        [18, 19]])\n",
      "tensor([[ 0],\n",
      "        [ 1],\n",
      "        [ 2],\n",
      "        [ 3],\n",
      "        [ 4],\n",
      "        [ 5],\n",
      "        [ 6],\n",
      "        [ 7],\n",
      "        [ 8],\n",
      "        [ 9],\n",
      "        [10],\n",
      "        [11],\n",
      "        [12],\n",
      "        [13],\n",
      "        [14],\n",
      "        [15],\n",
      "        [16],\n",
      "        [17],\n",
      "        [18],\n",
      "        [19]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(0, 20)\n",
    "\n",
    "print(x.view(1, 20))\n",
    "print(x.view(2, 10))\n",
    "print(x.view(4, 5))\n",
    "print(x.view(5, 4))\n",
    "print(x.view(10, 2))\n",
    "print(x.view(20, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use view to add size-1 dimensions, which can be useful for combining with other tensors.  This is called broadcasting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0,  1,  2,  3],\n",
      "        [ 4,  5,  6,  7],\n",
      "        [ 8,  9, 10, 11]])\n",
      "tensor([[0, 1, 2, 3]])\n",
      "tensor([[0],\n",
      "        [1],\n",
      "        [2]])\n",
      "tensor([[ 0,  2,  4,  6],\n",
      "        [ 4,  6,  8, 10],\n",
      "        [ 8, 10, 12, 14]])\n",
      "tensor([[ 0,  1,  2,  3],\n",
      "        [ 5,  6,  7,  8],\n",
      "        [10, 11, 12, 13]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(12).view(3, 4)\n",
    "y = torch.arange(4).view(1, 4)\n",
    "z = torch.arange(3).view(3, 1)\n",
    "\n",
    "print(x)\n",
    "print(y)\n",
    "print(z)\n",
    "print(x + y)\n",
    "print(x + z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsqueeze and squeeze will add and remove 1-dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 4])\n",
      "torch.Size([3, 1, 4])\n",
      "torch.Size([3, 4])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(12).view(3, 4)\n",
    "print(x.shape)\n",
    "\n",
    "x = x.unsqueeze(dim=1)\n",
    "print(x.shape)\n",
    "\n",
    "x = x.squeeze()\n",
    "print(x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "all of the standard mathematics operations apply (such as `add` below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: \n",
      " tensor([[0.6662, 0.3343, 0.7893, 0.3216],\n",
      "        [0.5247, 0.6688, 0.8436, 0.4265],\n",
      "        [0.9561, 0.0770, 0.4108, 0.0014]])\n",
      "--\n",
      "torch.add(x, x): \n",
      " tensor([[1.3324, 0.6686, 1.5786, 0.6433],\n",
      "        [1.0494, 1.3377, 1.6872, 0.8530],\n",
      "        [1.9123, 0.1540, 0.8216, 0.0028]])\n",
      "--\n",
      "x+x: \n",
      " tensor([[1.3324, 0.6686, 1.5786, 0.6433],\n",
      "        [1.0494, 1.3377, 1.6872, 0.8530],\n",
      "        [1.9123, 0.1540, 0.8216, 0.0028]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(3,4)\n",
    "print(\"x: \\n\", x)\n",
    "print(\"--\")\n",
    "print(\"torch.add(x, x): \\n\", torch.add(x, x))\n",
    "print(\"--\")\n",
    "print(\"x+x: \\n\", x + x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The convention of `_` indicating in-place operations continues:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0,  1,  2,  3],\n",
      "        [ 4,  5,  6,  7],\n",
      "        [ 8,  9, 10, 11]])\n",
      "tensor([[ 0,  2,  4,  6],\n",
      "        [ 8, 10, 12, 14],\n",
      "        [16, 18, 20, 22]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(12).reshape(3, 4)\n",
    "print(x)\n",
    "print(x.add_(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many operations for which reduce a dimension.  Such as sum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: \n",
      " tensor([[ 0,  1,  2,  3],\n",
      "        [ 4,  5,  6,  7],\n",
      "        [ 8,  9, 10, 11]])\n",
      "---\n",
      "Summing across rows (dim=0): \n",
      " tensor([12, 15, 18, 21])\n",
      "---\n",
      "Summing across columns (dim=1): \n",
      " tensor([ 6, 22, 38])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(12).reshape(3, 4)\n",
    "print(\"x: \\n\", x)\n",
    "print(\"---\")\n",
    "print(\"Summing across rows (dim=0): \\n\", x.sum(dim=0))\n",
    "print(\"---\")\n",
    "print(\"Summing across columns (dim=1): \\n\", x.sum(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Indexing, Slicing, Joining and Mutating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: \n",
      " tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "---\n",
      "x[:2, :2]: \n",
      " tensor([[0, 1],\n",
      "        [3, 4]])\n",
      "---\n",
      "x[0][1]: \n",
      " tensor(1)\n",
      "---\n",
      "Setting [0][1] to be 8\n",
      "tensor([[0, 8, 2],\n",
      "        [3, 4, 5]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(6).view(2, 3)\n",
    "print(\"x: \\n\", x)\n",
    "print(\"---\")\n",
    "print(\"x[:2, :2]: \\n\", x[:2, :2])\n",
    "print(\"---\")\n",
    "print(\"x[0][1]: \\n\", x[0][1])\n",
    "print(\"---\")\n",
    "print(\"Setting [0][1] to be 8\")\n",
    "x[0][1] = 8\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can select a subset of a tensor using the `index_select`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5],\n",
      "        [6, 7, 8]])\n",
      "---\n",
      "tensor([[0, 1, 2],\n",
      "        [6, 7, 8]])\n",
      "---\n",
      "tensor([[0, 2],\n",
      "        [3, 5],\n",
      "        [6, 8]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(9).view(3,3)\n",
    "print(x)\n",
    "\n",
    "print(\"---\")\n",
    "indices = torch.LongTensor([0, 2])\n",
    "print(torch.index_select(x, dim=0, index=indices))\n",
    "\n",
    "print(\"---\")\n",
    "indices = torch.LongTensor([0, 2])\n",
    "print(torch.index_select(x, dim=1, index=indices))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use numpy-style advanced indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 1, 2],\n",
      "        [6, 7, 8]])\n",
      "---\n",
      "tensor([[0, 1, 2],\n",
      "        [6, 7, 8]])\n",
      "---\n",
      "tensor([[0, 2],\n",
      "        [3, 5],\n",
      "        [6, 8]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(9).view(3,3)\n",
    "indices = torch.LongTensor([0, 2])\n",
    "\n",
    "print(x[indices])\n",
    "print(\"---\")\n",
    "print(x[indices, :])\n",
    "print(\"---\")\n",
    "print(x[:, indices])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine tensors by concatenating them.  First, concatenating on the rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([4, 3])\n",
      "Values: \n",
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5],\n",
      "        [0, 1, 2],\n",
      "        [3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 6])\n",
      "Values: \n",
      "tensor([[0, 1, 2, 0, 1, 2],\n",
      "        [3, 4, 5, 3, 4, 5]])\n",
      "Type: torch.LongTensor\n",
      "Shape/size: torch.Size([2, 2, 3])\n",
      "Values: \n",
      "tensor([[[0, 1, 2],\n",
      "         [3, 4, 5]],\n",
      "\n",
      "        [[0, 1, 2],\n",
      "         [3, 4, 5]]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(6).view(2,3)\n",
    "describe(x)\n",
    "describe(torch.cat([x, x], dim=0))\n",
    "describe(torch.cat([x, x], dim=1))\n",
    "describe(torch.stack([x, x]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can concentate along the first dimension.. the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5],\n",
      "        [6, 7, 8]])\n",
      "---\n",
      "torch.Size([3, 9])\n",
      "tensor([[0, 1, 2, 0, 1, 2, 0, 1, 2],\n",
      "        [3, 4, 5, 3, 4, 5, 3, 4, 5],\n",
      "        [6, 7, 8, 6, 7, 8, 6, 7, 8]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(9).view(3,3)\n",
    "\n",
    "print(x)\n",
    "print(\"---\")\n",
    "new_x = torch.cat([x, x, x], dim=1)\n",
    "print(new_x.shape)\n",
    "print(new_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also concatenate on a new 0th dimension to \"stack\" the tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5],\n",
      "        [6, 7, 8]])\n",
      "---\n",
      "torch.Size([3, 3, 3])\n",
      "tensor([[[0, 1, 2],\n",
      "         [3, 4, 5],\n",
      "         [6, 7, 8]],\n",
      "\n",
      "        [[0, 1, 2],\n",
      "         [3, 4, 5],\n",
      "         [6, 7, 8]],\n",
      "\n",
      "        [[0, 1, 2],\n",
      "         [3, 4, 5],\n",
      "         [6, 7, 8]]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(9).view(3,3)\n",
    "print(x)\n",
    "print(\"---\")\n",
    "new_x = torch.stack([x, x, x])\n",
    "print(new_x.shape)\n",
    "print(new_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear Algebra Tensor Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transposing allows you to switch the dimensions to be on different axis. So we can make it so all the rows are columsn and vice versa. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: \n",
      " tensor([[ 0,  1,  2,  3],\n",
      "        [ 4,  5,  6,  7],\n",
      "        [ 8,  9, 10, 11]])\n",
      "---\n",
      "x.tranpose(1, 0): \n",
      " tensor([[ 0,  4,  8],\n",
      "        [ 1,  5,  9],\n",
      "        [ 2,  6, 10],\n",
      "        [ 3,  7, 11]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(0, 12).view(3,4)\n",
    "print(\"x: \\n\", x) \n",
    "print(\"---\")\n",
    "print(\"x.tranpose(1, 0): \\n\", x.transpose(1, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A three dimensional tensor would represent a batch of sequences, where each sequence item has a feature vector.  It is common to switch the batch and sequence dimensions so that we can more easily index the sequence in a sequence model. \n",
    "\n",
    "Note: Transpose will only let you swap 2 axes.  Permute (in the next cell) allows for multiple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x.shape: \n",
      " torch.Size([3, 4, 5])\n",
      "x: \n",
      " tensor([[[ 0,  1,  2,  3,  4],\n",
      "         [ 5,  6,  7,  8,  9],\n",
      "         [10, 11, 12, 13, 14],\n",
      "         [15, 16, 17, 18, 19]],\n",
      "\n",
      "        [[20, 21, 22, 23, 24],\n",
      "         [25, 26, 27, 28, 29],\n",
      "         [30, 31, 32, 33, 34],\n",
      "         [35, 36, 37, 38, 39]],\n",
      "\n",
      "        [[40, 41, 42, 43, 44],\n",
      "         [45, 46, 47, 48, 49],\n",
      "         [50, 51, 52, 53, 54],\n",
      "         [55, 56, 57, 58, 59]]])\n",
      "-----\n",
      "x.transpose(1, 0).shape: \n",
      " torch.Size([4, 3, 5])\n",
      "x.transpose(1, 0): \n",
      " tensor([[[ 0,  1,  2,  3,  4],\n",
      "         [20, 21, 22, 23, 24],\n",
      "         [40, 41, 42, 43, 44]],\n",
      "\n",
      "        [[ 5,  6,  7,  8,  9],\n",
      "         [25, 26, 27, 28, 29],\n",
      "         [45, 46, 47, 48, 49]],\n",
      "\n",
      "        [[10, 11, 12, 13, 14],\n",
      "         [30, 31, 32, 33, 34],\n",
      "         [50, 51, 52, 53, 54]],\n",
      "\n",
      "        [[15, 16, 17, 18, 19],\n",
      "         [35, 36, 37, 38, 39],\n",
      "         [55, 56, 57, 58, 59]]])\n"
     ]
    }
   ],
   "source": [
    "batch_size = 3\n",
    "seq_size = 4\n",
    "feature_size = 5\n",
    "\n",
    "x = torch.arange(batch_size * seq_size * feature_size).view(batch_size, seq_size, feature_size)\n",
    "\n",
    "print(\"x.shape: \\n\", x.shape)\n",
    "print(\"x: \\n\", x)\n",
    "print(\"-----\")\n",
    "\n",
    "print(\"x.transpose(1, 0).shape: \\n\", x.transpose(1, 0).shape)\n",
    "print(\"x.transpose(1, 0): \\n\", x.transpose(1, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Permute is a more general version of tranpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x.shape: \n",
      " torch.Size([3, 4, 5])\n",
      "x: \n",
      " tensor([[[ 0,  1,  2,  3,  4],\n",
      "         [ 5,  6,  7,  8,  9],\n",
      "         [10, 11, 12, 13, 14],\n",
      "         [15, 16, 17, 18, 19]],\n",
      "\n",
      "        [[20, 21, 22, 23, 24],\n",
      "         [25, 26, 27, 28, 29],\n",
      "         [30, 31, 32, 33, 34],\n",
      "         [35, 36, 37, 38, 39]],\n",
      "\n",
      "        [[40, 41, 42, 43, 44],\n",
      "         [45, 46, 47, 48, 49],\n",
      "         [50, 51, 52, 53, 54],\n",
      "         [55, 56, 57, 58, 59]]])\n",
      "-----\n",
      "x.permute(1, 0, 2).shape: \n",
      " torch.Size([4, 3, 5])\n",
      "x.permute(1, 0, 2): \n",
      " tensor([[[ 0,  1,  2,  3,  4],\n",
      "         [20, 21, 22, 23, 24],\n",
      "         [40, 41, 42, 43, 44]],\n",
      "\n",
      "        [[ 5,  6,  7,  8,  9],\n",
      "         [25, 26, 27, 28, 29],\n",
      "         [45, 46, 47, 48, 49]],\n",
      "\n",
      "        [[10, 11, 12, 13, 14],\n",
      "         [30, 31, 32, 33, 34],\n",
      "         [50, 51, 52, 53, 54]],\n",
      "\n",
      "        [[15, 16, 17, 18, 19],\n",
      "         [35, 36, 37, 38, 39],\n",
      "         [55, 56, 57, 58, 59]]])\n"
     ]
    }
   ],
   "source": [
    "batch_size = 3\n",
    "seq_size = 4\n",
    "feature_size = 5\n",
    "\n",
    "x = torch.arange(batch_size * seq_size * feature_size).view(batch_size, seq_size, feature_size)\n",
    "\n",
    "print(\"x.shape: \\n\", x.shape)\n",
    "print(\"x: \\n\", x)\n",
    "print(\"-----\")\n",
    "\n",
    "print(\"x.permute(1, 0, 2).shape: \\n\", x.permute(1, 0, 2).shape)\n",
    "print(\"x.permute(1, 0, 2): \\n\", x.permute(1, 0, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matrix multiplication is `mm`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.4790,  0.8539, -0.2285],\n",
       "        [ 0.3081,  1.1171,  0.1585]], requires_grad=True)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.randn(2, 3, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 3])\n",
      "Values: \n",
      "tensor([[0., 1., 2.],\n",
      "        [3., 4., 5.]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([3, 2])\n",
      "Values: \n",
      "tensor([[1., 2.],\n",
      "        [1., 2.],\n",
      "        [1., 2.]])\n",
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[ 3.,  6.],\n",
      "        [12., 24.]])\n"
     ]
    }
   ],
   "source": [
    "x1 = torch.arange(6).view(2, 3).float()\n",
    "describe(x1)\n",
    "\n",
    "x2 = torch.ones(3, 2)\n",
    "x2[:, 1] += 1\n",
    "describe(x2)\n",
    "\n",
    "describe(torch.mm(x1, x2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.,  1.,  2.,  3.],\n",
      "        [ 4.,  5.,  6.,  7.],\n",
      "        [ 8.,  9., 10., 11.]])\n",
      "tensor([[1., 2.],\n",
      "        [1., 2.],\n",
      "        [1., 2.],\n",
      "        [1., 2.]])\n",
      "tensor([[ 6., 12.],\n",
      "        [22., 44.],\n",
      "        [38., 76.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.arange(0, 12).view(3,4).float()\n",
    "print(x)\n",
    "\n",
    "x2 = torch.ones(4, 2)\n",
    "x2[:, 1] += 1\n",
    "print(x2)\n",
    "\n",
    "print(x.mm(x2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the [PyTorch Math Operations Documentation](https://pytorch.org/docs/stable/torch.html#math-operations) for more!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing Gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[6., 9.]], grad_fn=<MulBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([[2.0, 3.0]], requires_grad=True)\n",
    "z = 3 * x\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this small snippet, you can see the gradient computations at work.  We create a tensor and multiply it by 3.  Then, we create a scalar output using `sum()`.  A Scalar output is needed as the the loss variable. Then, called backward on the loss means it computes its rate of change with respect to the inputs.  Since the scalar was created with sum, each position in z and x are independent with respect to the loss scalar. \n",
    "\n",
    "The rate of change of x with respect to the output is just the constant 3 that we multiplied x by."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: \n",
      " tensor([[2., 3.]], requires_grad=True)\n",
      "---\n",
      "z = 3*x: \n",
      " tensor([[6., 9.]], grad_fn=<MulBackward0>)\n",
      "---\n",
      "loss = z.sum(): \n",
      " tensor(15., grad_fn=<SumBackward0>)\n",
      "---\n",
      "after loss.backward(), x.grad: \n",
      " tensor([[3., 3.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([[2.0, 3.0]], requires_grad=True)\n",
    "print(\"x: \\n\", x)\n",
    "print(\"---\")\n",
    "z = 3 * x\n",
    "print(\"z = 3*x: \\n\", z)\n",
    "print(\"---\")\n",
    "\n",
    "loss = z.sum()\n",
    "print(\"loss = z.sum(): \\n\", loss)\n",
    "print(\"---\")\n",
    "\n",
    "loss.backward()\n",
    "\n",
    "print(\"after loss.backward(), x.grad: \\n\", x.grad)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Computing a conditional gradient\n",
    "\n",
    "$$ \\text{ Find the gradient of f(x) at x=1 } $$\n",
    "$$ {} $$\n",
    "$$ f(x)=\\left\\{\n",
    "\\begin{array}{ll}\n",
    "    sin(x) \\text{ if } x>0 \\\\\n",
    "    cos(x) \\text{ otherwise } \\\\\n",
    "\\end{array}\n",
    "\\right.$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    if (x.data > 0).all():\n",
    "        return torch.sin(x)\n",
    "    else:\n",
    "        return torch.cos(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.5403])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([1.0], requires_grad=True)\n",
    "y = f(x)\n",
    "y.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could apply this to a larger vector too, but we need to make sure the output is a scalar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "grad can be implicitly created only for scalar outputs",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-53-48e74398a3f8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0.5\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequires_grad\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/nlpbook/lib/python3.7/site-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    100\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m         \"\"\"\n\u001b[0;32m--> 102\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    103\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/nlpbook/lib/python3.7/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     82\u001b[0m         \u001b[0mgrad_tensors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad_tensors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 84\u001b[0;31m     \u001b[0mgrad_tensors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_grads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     85\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mretain_graph\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m         \u001b[0mretain_graph\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/nlpbook/lib/python3.7/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36m_make_grads\u001b[0;34m(outputs, grads)\u001b[0m\n\u001b[1;32m     26\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"grad can be implicitly created only for scalar outputs\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m                 \u001b[0mnew_grads\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: grad can be implicitly created only for scalar outputs"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([1.0, 0.5], requires_grad=True)\n",
    "y = f(x)\n",
    "# this is meant to break!\n",
    "y.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making the output a scalar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.5403, 0.8776])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([1.0, 0.5], requires_grad=True)\n",
    "y = f(x)\n",
    "y.sum().backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but there was an issue.. this isn't right for this edge case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-0.8415,  0.8415])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([1.0, -1], requires_grad=True)\n",
    "y = f(x)\n",
    "y.sum().backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.4794, 0.8415])\n"
     ]
    }
   ],
   "source": [
    "x = torch.tensor([-0.5, -1], requires_grad=True)\n",
    "y = f(x)\n",
    "y.sum().backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is because we aren't doing the boolean computation and subsequent application of cos and sin on an elementwise basis.  So, to solve this, it is common to use masking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.5403, 0.8415])\n"
     ]
    }
   ],
   "source": [
    "def f2(x):\n",
    "    mask = torch.gt(x, 0).float()\n",
    "    return mask * torch.sin(x) + (1 - mask) * torch.cos(x)\n",
    "\n",
    "x = torch.tensor([1.0, -1], requires_grad=True)\n",
    "y = f2(x)\n",
    "y.sum().backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def describe_grad(x):\n",
    "    if x.grad is None:\n",
    "        print(\"No gradient information\")\n",
    "    else:\n",
    "        print(\"Gradient: \\n{}\".format(x.grad))\n",
    "        print(\"Gradient Function: {}\".format(x.grad_fn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n",
      "No gradient information\n",
      "--------\n",
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([2, 2])\n",
      "Values: \n",
      "tensor([[21., 21.],\n",
      "        [21., 21.]], grad_fn=<AddBackward0>)\n",
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([])\n",
      "Values: \n",
      "21.0\n",
      "No gradient information\n",
      "--------\n",
      "Gradient: \n",
      "tensor([[2.2500, 2.2500],\n",
      "        [2.2500, 2.2500]], grad_fn=<CloneBackward>)\n",
      "Gradient Function: None\n",
      "--------\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "x = torch.ones(2, 2, requires_grad=True)\n",
    "describe(x)\n",
    "describe_grad(x)\n",
    "print(\"--------\")\n",
    "\n",
    "y = (x + 2) * (x + 5) + 3\n",
    "describe(y)\n",
    "z = y.mean()\n",
    "describe(z)\n",
    "describe_grad(x)\n",
    "print(\"--------\")\n",
    "z.backward(create_graph=True, retain_graph=True)\n",
    "describe_grad(x)\n",
    "print(\"--------\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.ones(2, 2, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AddBackward0 at 0x7f35ea134940>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.grad_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CUDA Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch's operations can seamlessly be used on the GPU or on the CPU.  There are a couple basic operations for interacting in this way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(torch.cuda.is_available())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.FloatTensor\n",
      "Shape/size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[0.9149, 0.3993, 0.1100],\n",
      "        [0.2541, 0.4333, 0.4451],\n",
      "        [0.4966, 0.7865, 0.6604]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(3,3)\n",
    "describe(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: torch.cuda.FloatTensor\n",
      "Shape/size: torch.Size([3, 3])\n",
      "Values: \n",
      "tensor([[0.1303, 0.3498, 0.3824],\n",
      "        [0.8043, 0.3186, 0.2908],\n",
      "        [0.4196, 0.3728, 0.3769]], device='cuda:0')\n",
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(3, 3).to(device)\n",
    "describe(x)\n",
    "print(x.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "cpu_device = torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "expected type torch.cuda.FloatTensor but got torch.FloatTensor",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-80-bc65a5d8cb7f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# this will break!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrand\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mx\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: expected type torch.cuda.FloatTensor but got torch.FloatTensor"
     ]
    }
   ],
   "source": [
    "# this will break!\n",
    "y = torch.rand(3, 3)\n",
    "x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.8394, 0.5273, 0.8267],\n",
       "        [0.9273, 1.2824, 1.0603],\n",
       "        [0.4574, 0.5968, 1.0541]])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = y.to(cpu_device)\n",
    "x = x.to(cpu_device)\n",
    "x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5274, 0.6325, 0.0910],\n",
      "        [0.2323, 0.7269, 0.1187],\n",
      "        [0.3951, 0.7199, 0.7595]], device='cuda:0')\n",
      "tensor([[0.5311, 0.6449, 0.7224],\n",
      "        [0.4416, 0.3634, 0.8818],\n",
      "        [0.9874, 0.7316, 0.2814]], device='cuda:0')\n",
      "tensor([[1.0585, 1.2775, 0.8134],\n",
      "        [0.6739, 1.0903, 1.0006],\n",
      "        [1.3825, 1.4515, 1.0409]], device='cuda:0')\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "expected type torch.FloatTensor but got torch.cuda.FloatTensor",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-82-0cfe18366dba>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# Error expected\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: expected type torch.FloatTensor but got torch.cuda.FloatTensor"
     ]
    }
   ],
   "source": [
    "if torch.cuda.is_available(): # only is GPU is available\n",
    "    a = torch.rand(3,3).to(device='cuda:0') #  CUDA Tensor\n",
    "    print(a)\n",
    "    \n",
    "    b = torch.rand(3,3).cuda()\n",
    "    print(b)\n",
    "\n",
    "    print(a + b)\n",
    "\n",
    "    a = a.cpu() # Error expected\n",
    "    print(a + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Exercises\n",
    "\n",
    "Some of these exercises require operations not covered in the notebook.  You will have to look at [the documentation](https://pytorch.org/docs/) (on purpose!)\n",
    "\n",
    "\n",
    "(Answers are at the bottom)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1\n",
    "\n",
    "Create a 2D tensor and then add a dimension of size 1 inserted at the 0th axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2\n",
    "\n",
    "Remove the extra dimension you just added to the previous tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 3\n",
    "\n",
    "Create a random tensor of shape 5x3 in the interval [3, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 4\n",
    "\n",
    "Create a tensor with values from a normal distribution (mean=0, std=1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 5\n",
    "\n",
    "Retrieve the indexes of all the non zero elements in the tensor torch.Tensor([1, 1, 1, 0, 1])."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 6\n",
    "\n",
    "Create a random tensor of size (3,1) and then horizonally stack 4 copies together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 7\n",
    "\n",
    "Return the batch matrix-matrix product of two 3 dimensional matrices (a=torch.rand(3,4,5), b=torch.rand(3,5,4))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 8\n",
    "\n",
    "Return the batch matrix-matrix product of a 3D matrix and a 2D matrix (a=torch.rand(3,4,5), b=torch.rand(5,4))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answers below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answers still below.. Keep Going"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Exercise 1\n",
    "\n",
    "Create a 2D tensor and then add a dimension of size 1 inserted at the 0th axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.rand(3,3)\n",
    "a = a.unsqueeze(0)\n",
    "print(a)\n",
    "print(a.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Exercise 2 \n",
    "\n",
    "Remove the extra dimension you just added to the previous tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = a.squeeze(0)\n",
    "print(a.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Exercise 3\n",
    "\n",
    "Create a random tensor of shape 5x3 in the interval [3, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "3 + torch.rand(5, 3) * 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Exercise 4\n",
    "\n",
    "Create a tensor with values from a normal distribution (mean=0, std=1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.rand(3,3)\n",
    "a.normal_(mean=0, std=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 5\n",
    "\n",
    "Retrieve the indexes of all the non zero elements in the tensor torch.Tensor([1, 1, 1, 0, 1])."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.Tensor([1, 1, 1, 0, 1])\n",
    "torch.nonzero(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 6\n",
    "\n",
    "Create a random tensor of size (3,1) and then horizonally stack 4 copies together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.rand(3,1)\n",
    "a.expand(3,4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 7\n",
    "\n",
    "Return the batch matrix-matrix product of two 3 dimensional matrices (a=torch.rand(3,4,5), b=torch.rand(3,5,4))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.rand(3,4,5)\n",
    "b = torch.rand(3,5,4)\n",
    "torch.bmm(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Exercise 8\n",
    "\n",
    "Return the batch matrix-matrix product of a 3D matrix and a 2D matrix (a=torch.rand(3,4,5), b=torch.rand(5,4))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.rand(3,4,5)\n",
    "b = torch.rand(5,4)\n",
    "torch.bmm(a, b.unsqueeze(0).expand(a.size(0), *b.size()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### END"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "nlpbook",
   "language": "python",
   "name": "nlpbook"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
